The FOCAL Point – Multimodal Dialogue with Virtual Geospatial Displays

The Future Operations Centre Analysis Laboratory (FOCAL) at Australia’s Defence Science and Technology Organisation (DSTO) is aimed at exploring new paradigms for situation awareness (SA) and command and control (C2) in military command centres, making use of new technologies developed for simulation, virtual reality, and real-time 3D animation. Recent work includes the development of a multimodal user interface to aid natural interaction with a virtual geospatial display using the combination of spoken natural language input and pointing devices. The end-user can query the environment, displayed objects, and their relationships using multimodal input and receive multimedia responses, optionally presented by a virtual adviser. The virtual geospatial display is a 3D representation of a geographical area of interest, rendered in real-time using imagery, maps, weather and terrain data, providing an augmented photo-realistic 3D model of the area. It allows the dynamic, near real-time display of objects within the environment, enhancing users’ situation awareness of the environment and of the relationships between the entities within it. FOCAL’s virtual advisers are real-time, photo-realistic embodied animated characters that converse with users using spoken language understanding, speech synthesis and facial gestures/cues. Pointing within the environment is performed with a variety of input devices that include the traditional mouse, 3D wands, gaze tracking and a novel hand gesture recognition system. FOCAL implements a heterogeneous multi-agent architecture based on the DARPA CoABS grid as a flexible, distributed model for interaction, information retrieval and processing, information fusion, and display. This paper provides an overview of FOCAL as a vision for future command centres, followed by a description of the latest developments of the multimodal dialogue system with virtual geospatial displays, illustrated in the context of a military training scenario.